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ABSTRACT 



The differential validity of subject area tests of 
academic ability is investigated. Principal components analyses of 
test scores, high school grades, and college grades in English math, 
social studies, and natural sciences show a dominant general ability 
dimension and a consistent configuration of subject areas on second 
and third dimensions. Data from approximately 250 colleges yield 
correlations of subject area college grades with subject area test 
scores on the American College Tests (ACT) and high school grades. A 
criterion of differential validity is proposed and calculated for the 
ACT tests and high school grades in predicting college grades. The 
moderate differential validity found is interpreted in terms of the 
first analysis. The modest differential validity in the ACT tests and 
minute amount in high school grades indicate that neither general 
tests of academic ability nor grades are ideally suited for efficient 
classification of students into one of several subject area 
curricula. (Author/KJ) 
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Summary 



The differential validity of subject area tests of academic ability is investigated. Principal components 

analyses of test scores, high school grades, and college grades in English, math, social studies, and natural 

sciences show a dominant general ability dimension and a consistent configuration of subject areas on 
second and third dimensions. 



Data from approximately 250 colleges yield correlations of subject area college grades with subject area 
test scores on the American College Tests and with high school grades. A criterion of differential validity 
is proposed and calculated for the ACT tests and high school grades in predicting college grades. The 
moderate differential validity found is interpreted in terms of the first analysis. 
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Differential Validity in the ACT Tests 



Nancy S. Cole 1 



Despite the successes of standardized tests of academic ability, one area has remained a problem. This is 
the area of differential prediction. The ease with which tests have predicted overall academic success has 
led to the demand for more specific tests to differentiate ability in various academic areas. 



Because of the persuasive content validity of many of these subject area tests, verification of their 
differential validity has too often been ignored. For example, in his review of the College Entrance 
Examination Board (CEEB) admissions testing program, Fricke (1965) criticized the Scholastic Aptitude 
Test (SAT) and the CEEB achievement tests for their lack of differential validity and also noted the 
relatively little research evidence available. 



The purpose of this paper is to investigate the differential validity of one commonly used college 
admissions test, the American College Test (ACT). Differential validity is of special concern because the 
relative scores on the four ACT tests in English, mathematics, social studies, and natural sciences are 
often used for evaluating a student's relative abilities in the four subject areas. 



Two important aspects of predictor, x, and criterion, y, behavior are related to differential validity. The 
first is the degree of the correlations among the variables within the predictor and criterion sets 
(Guilford, 1956; Thorndike, 1950; Wesman and Bennett, 1951; etc.). When these correlations, r x j ,xj an d 
r y j »y j ' are high, the predictors and criteria have little independent variance. Thus when Xj predicts y, xj 
also tends to predict it. Similarly, when yj is predicted by x, then yj also tends to be predicted by x. The 
relatedness of the ACT tests and high school grades, the predictors, and of college grades, the criteria, is 
considered in Study 1. 



A second and more direct indicator of differential validity comes from the comparison of the 
correlations r X( yj with rx^ for the set of predictors (Brogden, 1951; Cronbach, 1960; Horst, 1954; 
Mollenkopf, 1950; etc.). If x correlates positively with yj but little or negatively with y^, then x is a 
suitable differential predictor for yj and y^. In Study 2, thes.- i\y correlations are collected for both 
ACT tests and high school grades as x*and for college grades as y. 



Finally, in Study 3, a criterion of differential validity suitable to the differential use of test scores and 
high school grades is presented. Using data presented in the first two studies, we then evaluated the 
differential validity of the ACT tests and of high school grades according to the proposed criterion. 



*The author is indebted to James M. Richards, Jr. and Leo A. Munday for their helpful suggestions. 
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Study 1 

As already noted, differential prediction is limited by similarities among the criteria to be predicted and 
among the predictors. Thus, to evaluate and understand the amount of differential validity in tests of 
academic ability and in high school grades for predicting college grades differentially, we must first 
understand the degree of relatedness of the predictor variables and of the criteria. 

Data. The American College Testing Program provides research services to ACT-participating colleges. 
Included in the Standard Research Service analyses are correlations among the college grades in four 
subject areas (English— E, math— M, social studies— SS, and natural sciences— NS) which the colleges have 
reported. These correlations were collected for 
approximately 100 colleges participating in 1968 
with a combined N of over 20,000 for each 
correlation. The average of Fisher's z-transfor- 
mations weighted by their sample sizes, which 
were then transformed back to correlations, gave 
an estimate of the correlations among college 
grades. The correlations thus found are given in 
Table 1. 

Correlation matrices for self-reported HS grades 
(in E, M, SS, and NS) and for the four ACT tests 
(E, M, SS, NS) were available from Holland and Richards (1967) for a large (N=18, 378) representative 
sample of students taking the ACT test in 1964-65. These correlations are given in Table 2. A correlation 
matrix for the four ACT tests and the two SAT tests (Verbal-V and Math-M) was reported by 
Sassenrath and Pugh (1965) for 708 Indiana University students. 



Table 1 



Correlations Among College Subject Area Grades 

Col 

E M SS NS 

E ’ 



Col 



M 

SS 

NS 



- .43 .52 .50 

- .46 .54 

- .57 



Table 2 



Correlations* Among ACT Scores and High School Grades 





E 


M 


ACT 

SS 


NS 




E 


M 


HS 

SS 


NS 


E 


4.97 


.61 


.68 


.65 


E 


.85 


.44 


.56 


.48 


M 


.62 


6.38 


.61 


.62 


HS M 


.44 


.99 


.42 


.50 


SS 


.70 


.61 


6.28 


.75 


HS SS 


.55 


.43 


.90 


.48 


NS 


.66 


.61 


.74 


6.09 


NS 


.46 


.46 


.49 


.92 



•/Vofe-Correlations for men are above the diagonals and for women below. Standard deviations for men are on 
the diagonals. (From Holland and Richards, 1966, p. 5) 
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Results. To understand the similarities and differences in the four subject areas (E,M,SS,and NS), we 
submitted each of the four correlation matrices available (college grades, high school grades, ACT scores, 
and ACT plus SAT scores) to a principal components analysis. The results were strikingly similar in all 
four cases. The first root was quite dominant, accounting for from 61% to 74% of the trace. The 
corresponding loadings were high and positive for all tests or grades (from .74 to .89) indicating many 
abilities (or one general ability) common to the subject areas. 




FIG. 1. Subject areas in college grades, high school 
grades, and tests are plotted on second and third 
dimensions from principal components analyses. 



However, the dimensions on which the remaining 
small differences existed could be important to the 
question of differential prediction. The configuration 
of the subject areas on the second and third dimen- 
sions in the principal components analyses are given 
in Figure 1. All four analyses yielded points within 
the regions shown. The points for the Sassenrath and 
Pugh matrix are given to relate the ACT and SAT 
tests. The reliability of the configuration of subject 
areas is confirmed by its occurrence in tests, college 
grades, and high school grades and from data of 
different sources. 

Discussion. From these analyses we know that the 
tests and the grades they are to predict have sizable 
similarities. However, consistent though moderately- 



sized differences in high school and college subject area grades suggest that some differential prediction 
can occur. 



Study 2 

To directly assess the differential validity of the ACT tests (and of high school grades for comparison), 
we obtained the correlations of college grades in the four subject areas (E, M, SS, and NS) with the four 
ACT tests and high school grades. 
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Data. Part of the analyses provided colleges by the American College Testing Program research services 
is the correlations of the ACT scores and self-reported high school grades (routinely reported by the 
student in the ACT battery) with first semester college grades. For the years 1966 and 1967 the 
correlations for the approximately 250 colleges with this data were collected. Not all colleges reported 
grades for all four areas so the numbers in each area differ as follows: E— 278 colleges (N=229, 265); 
M— 226 colleges (N=80, 945); SS-261 colleges (N=141, 651); and NS-235 colleges (N=133,702). 
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Results. The averages of Fisher's z-transformations weighted by the sample size and then transformed 
back to correlations are given in Table 3. The correlations are lower than would ordinarily be found in 
an unselected group because of some restriction of range within the colleges, but this should not affect 
the comparisons made within the table. 



Table 3 



Correlations of College Grades with ACT Tests and High School Grades 





ACT 

E 


ACT 

M 


ACT 

SS 


ACT 

NS 


HS 

E 


HS 

M 


HS 

SS 


HS 

NS 


COL-E 


.47 


.24 


.34 


.27 


.43 


.26 


.30 


.27 


COL-M 


.28 


.38 


.24 


.22 


.30 


.35 


.29 


.30 


COL-SS 


.34 


.29 


.43 


.33 


.37 


.28 


.40 


.31 


COL-NS 


.35 


.39 


.38 


.39 


.37 


.37 


.37 


.37 



i 
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Discussion. The amount df differential validity in the ACT scores and in high school grades is indicated 
by a comparison of a diagonal correlation in each 4x4 matrix with correlations in the column in which it 
lies. For example, the correlation between ACT— M and COL— M is .38. In the column of ACT— M we 
find as high a correlation with COL-NS as with COL-M (.39 to .38). Thus, ACT-M gives no 
differentiation between COL-M and COL-NS although it does differentiate between quantitative and 
nonquantitative fields. 



Table 3 clearly indicates that the ACT tests show as much if not more differential validity than do high 
school grades. In both cases the amount of differential prediction appears to be moderate. 



Study 3 



A criterion is needed to make explicit the evaluation of differential prediction. As used here differential 
prediction refers to the ability of predictors to predict differences in criteria. Thus a reasonable criterion 
of the differential validity of a pair of predictors (xj and Xj) for a pair of college grades (y k and y h ) is the 
correlation of the difference d x = xj — xj with dy = y k — y^. This correlation indicates the degree to 
which a difference in the predictors is related to a difference in the criteria. In the case of the ACT tests, 



for example, if a student's English score is higher than his math score, this correlation indicates the 
degree to which we can expect his college English grade to be higher than his grade in college math. 
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The correlation of differences, rd x ,d y , can be expressed algebraically in terms of the variances and 



covariances of the original four variables: 



r d x ,d y - 



s Xj,y k + Sx j.y h ~ S xj.Vh ~ Sx j'Vk 



( 1 ) 



for 



~> s J"(Sxj,xj + S X j, Xj ~ 2S X j,xj) (Sy k ,y k + Sy^y^ 



N 



S u ,v = 2 (u q - u) (v q - v). 

q=1 



(2) 



The meaning of (1) becomes clear when S Xj(Xj = S Xj -,xj and s y k ,y k ” s Yh<yh' a case whlch ls 
approximated with the ACT tests, high school grades, and college grades. Then 



r d x »dy = 



r xj//k + rx j'Vh r xj,yh rx j.Vk 



(3) 



N 



2 (1 -r Xj ,Xj) n -ry k ,y h ) 



The subscripts in (1) and (3) are written generally to allow for calculation of the differential validity of 
ACT (vSS-NS) for predicting college (E-M), for example. However, in practice, only the corresponding 
subject areas are usually used. 



It should be noted that when y k and y^ are replications of xj and xj (the same tests given at a later time, 
say), then calling the later tests xj> and x:* and the difference d x ',rd X( d x < * s a measure of the reliability of 



difference scores. Letting r XjfX '. = r x ' j( x'i = r Xj ,xj. reduces (3) to a familiar form (Gulliksen, 1950), 



i' j 



M' J 



r d X ' d x' = 



( r xj,x'j + r xj,x'j) / 2 r X j, X j 



(4) 



(1 - r x j,Xj ) 



Although our concern is with differential validity, we can appropriately be reminded of the reliability of 
the differential predictors. 



Data. In order to calculate rd x ,d y for pairs of ACT tests or high school grades and college grades by (3), 
only the standard deviations of college grades are needed to supplement the data already presented. 
From a sample of colleges in Study 2 we determined that the standard deviations of college grades 











within colleges were quite similar to those across colleges. Thus the standard deviations of college grades 
for students at institutions participating in ACT Research Services in 1965, 1966, and 1967 were used. 
These standard deviations were .96, 1.18, 1.02, and 1.07 for college E, M, SS, and NS, respectively 
(Hoyt and Munday, 1968, p. 205). 



Results. Although every pair of predictors is a potential differential predictor for a pair of college 
grades, usually the corresponding pairs (predictors E, M for criteria E, M) were the best. The correlations 
of differences between predictors with differences between criteria calculated using (3) are reported in 
Table 4 for ACT scores and in Table 5 for high school grades. The correlations on the diagonals are put 
in parentheses to identify them as the differential validity of the predictor variables corresponding to the 
same subject areas as those being predicted. When an off-diagonal validity is greater that the diagonal in 
its row or column, it is underlined. 
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Table 4 



Correlations of ACT Score Differences with College Grade Differences 

ACT 







E-M 


E-SS 


E-NS 


M-SS 


M-NS 


SS-NS 




E-M 


(.33) 


.06 


.11 


-.26 


-.22 


.05 




E-SS 


.19 


(.27) 


.21 


.05 


.01 


-.05 


College 


E-NS 


,29 


.18 


(.27) 


-.12 


-.04 


.10 


M-SS 


-.18 


.16 


.07 


(.30) 


.23 


-.10 




M-NS 


-.07 


.11 


.14 


.16 


(.20) 


.04 




SS-NS 


.12 


-.07 


.08 


-.18 


-.05 


(.16) 



Table 5 



correlations of High School Grade Differences with College Grade Differences 

High School 





E-M 


E-SS 


E-NS 


M-SS 


M-NS 


SS-NS 


E-M 


(.19) 


.11 


.12 


-.09 


-.07 


.03 


E-SS 


.06 


(.18) 


.09 


.08 


.02 


-.07 


E-NS 


.16 


.14 


(.15) 


-.04 


-.02 


.02 


M-SS 


-.13 


.04 


-.05 


(.16) 


J09 


-.08 


M-NS 


-.05 


.02 


.01 


.06 


(.06) 


-.01 


SS-NS 


.10 


-.02 


.07 


-.12 


-.04 


(.09) 
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Discussion. As suggested in Study 2, the amount of differential validity in ACT tests and high school 
grades is small. However, it is interesting to note the differences among the combinations. The E, M 
difference is predicted best, and two other verbal-quantitative differences (E, NS and M, SS) rank high. 
SS, NS probably ranks low because both the ACT NS test and high school NS are not as mathematical as 
is college NS. (ACT SS and NS tests are both largely nonquantitative reading tests.) This possibility is 
confirmed by the fact that SS-M is a slightly better predictor of college SS-NS for both ACT scores 
and high school grades. A similar situation seems to exist in regard to the lowM-NS validity. None of 
the tests or high school grades correlate better with college NS than any of the others do, and, in 
particular, ACT NS and high school NS are probably related to college NS mainly through the general 
ability factor rather than uniquely. In this connection note that E-M predicts college E-NS slightly 

better than E-NS. 

Finally, it should be noted that high school grades are especially poor differential predictors. In addition, 
the reliability of the difference scores of both test scores and high school grades is low. 



Conclusions 

This report began by noting the general successes of academic ability testing. From the results presented 
here it seems likely that the reasons for those successes are also the reasons for the difficulty in 
differential prediction. Many areas of academic endeavor involve abilities in common as seen in the first 
principal component in high school grades, college grades, and test scores. It is these common abilities 
that are largely measured by tests of academic ability. When differential prediction is desired, the 
proportion of variation in the predictors and criteria which is unique becomes important. Since the 
criteria have only a small proportion of variance unique to different subject areas, a test tapping that 
uniqueness, though suited for differential prediction, would probably have low general validity. 

The modest differential validity in the ACT tests and minute amount in high school grades indicate that 
neither general tests of academic ability nor grades are ideally suited for efficient classification of 
students into one of several subject area curricula. It seems likely that with special consideration given to 
differential validity in the construction of the tests, differential prediction could be somewhat improved. 
However in tests measuring abilities dominant in college courses, probably only moderate differentia! 
validity can be attained. In fact, the goals of general validity and differential validity are apparently in 

direct conflict in tests of academic ability. 
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